Remote monitoring system, remote monitoring method, and remote monitoring server

ABSTRACT

The on-board information processing apparatus executes: detecting an object ahead of the vehicle from an image acquired by an on-board camera; compressing the image to generate a compressed image; transmitting first data including the compressed image and an image acquisition time of the image before compression; and transmitting second data including an object detection result and an image acquisition time of the image used for object detection. The remote monitoring server executes: receiving the first data to store it in a memory; receiving the second data to store it in the memory; extracting the compressed image and the object detection result whose image acquisition times are the same time from the memory in time series; and superimposing an extracted object detection result on a restored image acquired by restoring an extracted compressed image to display the restored image on which the object detection result is superimposed on a monitoring screen.

CROSS-REFERENCE TO RELATED APPLICATION

The present application claims priority under 35 U.S.C. § 119 toJapanese Patent Application No. 2021-064382, filed Apr. 5, 2021, thecontents of which application are incorporated herein by reference intheir entirety.

BACKGROUND Field

The present disclosure relates to a technique for remotely monitoring avehicle.

Background Art

JP2020-161039A discloses a prior art related to a remote control devicefor remotely controlling an autonomous traveling vehicle which hasbecome impossible to perform autonomous traveling. The remote controldevice according to the prior art, when displaying an image taken by acamera mounted on the autonomous traveling vehicle on a display unit,analyzes the taken image and highlights a specific object on the image.

The image taken by the camera is compressed and then transmitted fromthe autonomous traveling vehicle to the remote control device. Althoughcompression of the image is indispensable for reducing a communicationcost, the resolution of the image is lowered by being compressed.Therefore, in the above-mentioned prior art, the image cannot beanalyzed with high accuracy, and highlighting of a specific object maynot be appropriately performed.

As prior art documents representing the technical level of the technicalfield to which the present disclosure belongs, in addition toJP2020-161039A, JP2018-56838A and JP2011-96063A can be exemplified.

SUMMARY

The present disclosure has been made in view of the above-describedproblems. An object of the present disclosure is to provide a techniquecontributing to improvement of accuracy of a detection result of anobject displayed on a monitor screen while reducing a communicationcost.

The present disclosure provides a remote monitoring system. The remotemonitoring system according to the present disclosure includes a cameramounted on a vehicle, an on-board information processing apparatusmounted on the vehicle and connected to the camera, and a remotemonitoring server connected to the on-board information processingapparatus via a communication network. The on-board informationprocessing apparatus executes an object detection process, a compressionprocess, a first data transmission process, and a second datatransmission process as mentioned below. The remote monitoring serverexecutes a first data reception process, a second data receptionprocess, an extraction process, and a display process as mentionedbelow.

In the object detection process by the on-board information processingapparatus, it is executed to detect an object ahead of the vehicle froman image acquired by the camera. In the compression process, the imageis compressed to generate a compressed image. In the first datatransmission process, first data including the compressed image and animage acquisition time of the image before compression is transmitted tothe remote monitoring server. In the second data transmission process,second data including an object detection result and an imageacquisition time of the image used for object detection is transmittedto the remote monitoring server.

In the first data reception process by the remote monitoring server, thefirst data transmitted from the on-board information processingapparatus is received and stored in a memory. In the second datareception process, the second data transmitted from the on-boardinformation processing apparatus is received and stored in the memory.In the extraction process, the compressed image and the object detectionresult whose image acquisition times are the same time are extractedfrom the memory in time series. In the display process, an extractedobject detection result is superimposed on a restored image acquired byrestoring an extracted compressed image, and the restored image on whichthe extracted object detection result is superimposed is displayed on amonitoring screen.

In the present remote monitoring system, the on-board informationprocessing apparatus may further execute third data transmissionprocess, and the remote monitoring server may further execute third datareception process. In the third data transmission process, it isexecuted to transmit the third data including sensor informationacquired from a sensor signal of a on board sensor and a signalacquisition time of the sensor signal. In the third data receptionprocess, the third data transmitted from the on-board informationprocessing apparatus is received and stored in the memory. In this case,in the extraction process by the remote monitoring server, thecompressed image and the object detection whose image acquisition timesare the same time are extracted from the memory in time series, and thesensor information whose signal acquisition time is the same time as theimage acquisition times is extracted from the memory in time series. Inthe display process, an extracted sensor information is superimposed onthe restored image together with the extracted object detection result,and the restored image on which the extracted object detection resultand the extracted sensor information are superimposed is displayed onthe monitoring screen.

Also, the present disclosure provides a remote monitoring method. Theremote monitoring method according to the present disclosure is a methodof remotely monitoring a vehicle using an on-board informationprocessing apparatus and a remote monitoring server connected via acommunication network. The remote monitoring method causes the on-boardinformation processing apparatus to execute detecting an object ahead ofthe vehicle from an image acquired by a camera mounted on the vehicle,and compressing the image to generate a compressed image. The remotemonitoring method causes the on-board information processing apparatusto further execute transmitting first data including the compressedimage and an image acquisition time of the image before compression, andtransmitting second data including an object detection result and animage acquisition time of the image used for object detection. Further,the remote monitoring method causes the remote monitoring server toexecute receiving the first data to store the first data in a memory,and receiving the second data to store the second data in the memory.The remote monitoring method causes the remote monitoring server tofurther execute extracting the compressed image and the object detectionresult whose image acquisition times are the same time from the memoryin time series, and superimposing an extracted object detection resulton a restored image acquired by restoring an extracted compressed imageto display the restored image on which the extracted object detectionresult is superimposed on a monitoring screen.

The remote monitoring method may cause the on-board informationprocessing apparatus to execute transmitting third data including sensorinformation acquired from a sensor signal of an on-board sensor and asignal acquisition time of the sensor signal. The remote monitoringmethod may cause the remote monitoring server to execute receiving thethird data to store the third data in the memory, extracting thecompressed image and the object detection whose image acquisition timesare the same time from the memory in time series, and extracting thesensor information whose signal acquisition time is the same time as theimage acquisition times from the memory in time series. The remotemonitoring method may cause the remote monitoring server to furtherexecute superimposing an extracted sensor information on the restoredimage together with the extracted object detection result to display therestored image on which the extracted object detection result and theextracted sensor information are superimposed on the monitoring screen.

Further, the present disclosure provides a remote monitoring server. Theremote monitoring server according to the present disclosure is a serverconnected to an on-board information processing apparatus via acommunication network. The remote monitoring server includes at leastone memory storing at least one program and at least one processorcoupled to the at least one memory. The at least one program isconfigured to cause the at least one processor to perform the followingprocess.

In processing executed by the at least one processor of the remotemonitoring server, first data including a compressed image generated bycompressing an image acquired by an on-board camera and an imageacquisition time of the image before compression is received from theon-board information processing apparatus and stored in the at least onememory. In addition, second data including an object detection resultbased on the image and an image acquisition time of the image used forthe object detection is received from the on-board informationprocessing apparatus and stored in the at least one memory. Further, thecompressed image and the object detection result whose image acquisitiontimes are the same time are extracted from the at least one memory intime series. Further, an extracted object detection result issuperimposed on a restored image acquired by restoring an extractedcompressed image, and the restored image on which the extracted objectdetection result is superimposed is displayed on a monitoring screen.

Further, the present disclosure provides an on-board informationprocessing apparatus. The on-board information processing apparatusaccording to the present disclosure is an apparatus connected to aremote monitoring server via a communication network. The on-boardinformation processing apparatus includes at least one memory storing atleast one program, and at least one processor coupled to the at leastone memory. The at least one program is configured to cause the at leastone processor to perform the following processing.

In the processing executed by the at least one processor of the presenton-board information processing apparatus, detecting an object ahead ofa vehicle from an image acquired by an on-board camera is executed.Further, a compressed image is generated by compressing the image. Firstdata including the compressed image and an image acquisition time of theimage before compression is transmitted to the remote monitoring server.Further, second data including an object detection result and an imageacquisition time of the image used for object detection is transmittedto the remote monitoring server.

According to the present disclosure, since an object ahead of a vehicleis detected from an image before compression, it is possible to preventdeterioration of detection accuracy compared to the case where theobject is detected from a compressed image. This contributes toimprovement of accuracy of a detection result of an object displayed ona monitor screen while reducing a communication cost.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram schematically illustrating a configuration of aremote monitoring system according to embodiments of the presentdisclosure.

FIG. 2 is an example of an image before compression for explaining aproblem of a remote monitoring system.

FIG. 3 is an example of an object detection result from an image beforecompression for explaining a problem of a remote monitoring system.

FIG. 4 is an example of an image after compression for explaining aproblem of a remote monitoring system.

FIG. 5 is an example of an object detection result from an image aftercompression for explaining a problem of a remote monitoring system.

FIG. 6 is a diagram for explaining an outline of features of the remotemonitoring system according to the embodiments of the presentdisclosure.

FIG. 7 is a block diagram showing a configuration of a vehicle accordingto a first embodiment of the present disclosure.

FIG. 8 is a block diagram showing a configuration of a remote monitoringcenter according to the first embodiment of the present disclosure.

FIG. 9 is a block diagram showing processing executed by each processorof an on-board information processing apparatus and a remote monitoringserver according to the first embodiment of the present disclosure.

FIG. 10 is a block diagram showing a configuration of a vehicleaccording to a second embodiment of the present disclosure.

FIG. 11 is a block diagram showing a configuration of a remotemonitoring center according to the second embodiment of the presentdisclosure.

FIG. 12 is a block diagram showing processing executed by each processorof an on-board information processing apparatus and a remote monitoringserver according to the second embodiment of the present disclosure.

DETAILED DESCRIPTION

Hereunder, embodiments of the present disclosure will be described withreference to the drawings. Note that when the numerals of numbers,quantities, amounts, ranges and the like of respective elements arementioned in the embodiments shown as follows, the present disclosure isnot limited to the mentioned numerals unless specially explicitlydescribed otherwise, or unless the disclosure is explicitly designatedby the numerals theoretically. Furthermore, structures that aredescribed in the embodiments shown as follows are not alwaysindispensable to the disclosure unless specially explicitly shownotherwise, or unless the disclosure is explicitly designated by thestructures or the steps theoretically.

1. Schematic Configuration of Remote Monitoring System

FIG. 1 is a diagram schematically illustrating a configuration of aremote monitoring system according to embodiments of the presentdisclosure. The remote monitoring system 2 is a system for remotelymonitoring a vehicle 10 traveling on a road by a remote operator 50.Remote monitoring in the present disclosure includes monitoring forremotely driving the vehicle 10, monitoring for remotely supporting thevehicle 10 traveling autonomously, according to the situation, andmonitoring for confirming the surrounding environment of the vehicle 10traveling autonomously.

The vehicle 10 to be remotely monitored is equipped with a camera 12.The camera 12 may be a stereo camera or a monocular camera. The imagingarea IA of the camera 12 covers at least an area ahead of the vehicle 10in the traveling direction. For example, if the vehicle 10 is travelingin the traveling lane DL, the area in front of the traveling lane DL andareas on both left and right sides thereof are included in the imagingarea IA of the camera 12.

The vehicle 10 is equipped with an on-board information processingapparatus 20. An image taken by the camera 12 is input to the on-boardinformation processing apparatus 20. The on-board information processingapparatus 20 is connected to a remote monitoring server 40 of the remotemonitoring center 30 via a communication network 8. The communicationnetwork 8 includes a mobile communication network such as 4G and 5G. Theimage taken by the camera 12 is sent from the on-board informationprocessing apparatus 20 to the remote monitoring server 40 via thecommunication network 8.

In the remote monitoring center 30, the image of the camera 12 sent fromthe on-board information processing apparatus 20 to the remotemonitoring server 40 is displayed on a monitoring screen 32 a of adisplay device 32. As the display device 32, for example, a liquidcrystal display, an organic EL display, a head mount display, a touchpanel can be used. The remote operator 50 can monitor ahead of thevehicle 10 by the image of the camera 12 projected on the monitoringscreen 32 a of the display device 32. For example, if an obstacle OJ ispresent in the imaging area IA of the camera 12, the remote operator 50can recognize the presence of the obstacle OJ in the monitoring screen32 a of the display device 32.

The remote monitoring system 2 has a function of assisting the remoteoperator 50 to recognize the obstacle OJ. The obstacle OJ in the imageof the camera 12 can be automatically detected by using an imageanalysis technique such as pattern matching and deep learning. Byspecifying the obstacle OJ in the image using this object detectionfunction, it is possible to inform the remote operator 50 where theobstacle OJ exists in the monitor screen 32 a.

On the other hand, the remote monitoring system 2 is required to reducecommunication cost. The image taken by the camera 12 is encoded by theon-board information processing apparatus 20, and the encoded image datais transmitted to the remote monitoring server 40. In the encoding, theimage taken by the camera 12 is compressed by a predeterminedcompression scheme. The image data transmitted from the on-boardinformation processing apparatus 20 to the remote monitoring server 40is decoded by the remote monitoring server 40. An image restored bydecoding by the remote monitoring server 40 is displayed on the displaydevice 32.

It should be noted that there is no limitation on the image transmissionformat, and there is no limitation on the image compression format. Whenthe image transmission format is a moving image stream format, forexample, H.264, H.265, VP8, VP9, VP10, and AV1 can be used as the imagecompression format. When the image transmission format is a format inwhich transmission is performed image by image, for example, JPEG,Motion JPEG and PNG can be used as the image compression format.Preferably, H.264, VP8, or VP9 is used. Motion JPEG, VP10 and AV1 arealso suitable compression formats for the remote monitoring system 2.

2. Problem of Remote Monitoring System

As described above, the remote monitoring system 2 is required to assistthe remote operator 50 by object detection from the image taken by thecamera 12, and reduce communication cost by compression of the imagetaken by the camera 12. However, it is not easy to satisfy these tworequirements simultaneously.

Here, four images are shown. FIG. 2 is an example of an original imagetaken by the camera 12, that is, an image before compression. FIG. 3 isan example of an object detection result from the original image shownin FIG. 2 . FIG. 4 is an example of an image acquired by compressing theoriginal image shown in FIG. 2 , that is, an image after compression.FIG. 5 is an example of an object detection result from the image aftercompression shown in FIG. 4 . However, these images are originally colorimages, but are binarized in these figures for the convenience of theapplication documents.

The image displayed on the monitor screen 32 a of the display device 32is the image after compression shown in FIG. 4 . However, the resolutionof the image after compression is lower than that of the original image.This is also evident from the binarized image, as can be seen from acomparison of the image shown in FIG. 2 with the image shown in FIG. 4 .Therefore, as shown in FIG. 3 , two persons and two vehicles aredetected from the original image with high resolution, whereas twovehicles cannot be detected from the compressed image, as shown in FIG.5 .

As in the above example, an object to be detected may not be detectedfrom the compressed image. In addition, although not in the aboveexample, a non-existing object may be erroneously detected from thecompressed image. That is, the image transmitted from the on-boardinformation processing apparatus 20 to the remote monitoring server 40needs to be compressed, but the object ahead of the vehicle 10 cannot bedetected with high accuracy from the compressed image.

3. Outline of Features of Remote Monitoring System

According to the remote monitoring system 2, the above-mentioned problemis solved as follows. FIG. 6 is a diagram for explaining an outline offeatures of the remote monitoring system 2.

FIG. 6 shows the flow of information in the remote monitoring system 2.In the remote monitoring system 2, a compressed image is acquired froman original image before compression by the on-board informationprocessing apparatus 20. In parallel with this, an object detectionresult is acquired from the original image before compression by theon-board information processing apparatus 20. The compressed image andthe object detection result acquired separately are transmittedseparately from the on-board information processing apparatus 20 to theremote monitoring server 40 via the communication network 8. That is, inthe remote monitoring system 2, the process of detecting the object fromthe image of the camera 12 is performed in the on-board informationprocessing apparatus 20 before the transmission of the image to theremote monitoring server 40, and is performed not on the compressedimage but on the original image before compression.

The compressed image transmitted from the on-board informationprocessing apparatus 20 to the remote monitoring server 40 is restoredby the remote monitoring server 40. The object detection resulttransmitted from the on-board information processing apparatus 20separately from the compressed image is superimposed on the restoredimage by the remote monitoring server 40. As a result, a superimposeddisplay screen in which the object detection result is superimposed onthe restored image is acquired. As a specific example, the image aftercompression shown in FIG. 4 is displayed on the monitor screen 32 a ofthe display device 32, and the object detection result shown in FIG. 3is superimposed on the image.

According to the remote monitoring system 2 having the above features,by performing the object detection with respect to the original imagebefore compression by the on-board information processing apparatus 20,it is possible to detect the object ahead of the vehicle 10 with highaccuracy. By displaying the object detection result together with thecompressed image on the monitor screen 32 a of the display device 32,remote monitoring by the remote operator 50 can be assisted well.

Although compression process of the original image and object detectionprocess from the original image are included in processing executed bythe on-board information processing apparatus 20, they are executedseparately. Therefore, even if the original images are common, there maybe a difference between the time at which the compressed image isacquired and the time at which the object detection result is acquired.A time difference between the compressed image and the object detectionresult may mislead the remote operator 50.

The following chapters describe the detailed configuration and functionsof the remote monitoring system 2. As described therein, the remotemonitoring system 2 is technically devised so as not to cause a timedifference between an image and an object detection result that aredisplayed on the display device 32.

4. Configuration of Vehicle According to First Embodiment

FIG. 7 is a block diagram showing a configuration of a vehicle 10equipped with an on-board information processing apparatus 20 accordingto the first embodiment. The vehicle 10 comprises the on-boardinformation processing apparatus 20, and further comprises a camera 12,an on-board sensor 14, a communication device 16, and a traveling system18. The details of the camera 12 mounted on the vehicle 10 are asdescribed above.

The on-board sensor 14 includes a state sensor for acquiring informationabout the motion of the vehicle 10. The state sensor includes, forexample, at least one of a speed sensor, an acceleration sensor, a yawrate sensor, and a steering angle sensor. Further, the on-board sensor14 includes a position sensor for detecting the position and orientationof the vehicle 10. As the position sensor, a GPS (Global PositioningSystem) sensor is exemplified. Further, the on-board sensor 14 mayinclude a recognition sensor other than the camera 12. The recognitionsensor recognizes the surrounding condition of the vehicle 10. Examplesof the recognition sensor include a LiDAR (Laser Imaging Detection andRanging), a millimeter wave radar, and an ultrasonic sensor.

The communication device 16 communicates with the outside of the vehicle10. The communication device 16 communicates with the remote monitoringserver 40 via the communication network 8.

The traveling system 18 includes a steering system for steering thevehicle 10, a driving system for driving the vehicle 10, and a brakingsystem for braking the vehicle 10. The steering system includes, forexample, a power steering system, a steer-by-wire steering system, and arear wheel steering system. The driving system includes, for example, anengine system, an EV system, and a hybrid system. The braking systemincludes, for example, a hydraulic brake and a power regenerative brake.

The on-board information processing apparatus 20 is an ECU (ElectronicControl Unit) mounted on the vehicle 10 or an aggregate of a pluralityof ECUs. The on-board information processing apparatus 20 includes atleast one processor 21 (hereinafter simply referred to as a processor21) and at least one memory 22 (hereinafter simply referred to as amemory 22) coupled to the processor 21. The memory 22 includes a mainstorage and an auxiliary storage. The memory 22 stores at least oneprogram (hereinafter simply referred to as a program) executable by theprocessor 21 and various related information. When the processor 21executes the program, various kinds of processing by the processor 21are realized. The program may be stored in the main storage or in acomputer-readable recording medium that is the auxiliary storage.

At least three storage areas 22 a, 22 b, and 22 c are prepared in thememory 22. The processor 21 stores pre-processing data DT0 in the firststorage area 22 a of the memory 22. The pre-processing data DT0 includesan image IMG taken by the camera 12 and a time stamp TS indicating thetime at which the image IMG was acquired. The processor 21 reads out thepre-processing data DT0 and executes the compression process.Furthermore, the processor 21 reads out the pre-processing data DT0 andexecutes the object detection process.

The processor 21 stores first processing data (hereinafter, referred toas first data) DT1 in the second storage area 22 b of the memory 22. Thefirst data DT1 includes a compressed image CIMG acquired by thecompression process for the image IMG and the time stamp TS associatedwith the image IMG before compression. The processor 21 uses thecommunication device 16 to transmit the first data DT1 to the remotemonitoring server 40.

The processor 21 stores second processing data (hereinafter, referred toas second data) DT2 in the third storage area 22 c of the memory 22. Thesecond data DT2 includes an object detection result ODR acquired by theobject detection process for the image IMG and the time stamp TSassociated with the image IMG used for object detection. The processor21 uses the communication device 16 to transmit the second data DT2 tothe remote monitoring server 40.

5. Configuration of Remote Monitoring Center According to FirstEmbodiment

FIG. 8 is a block diagram showing a configuration of the remotemonitoring center 30 that includes the remote monitoring server 40according to the first embodiment. The remote monitoring center 30includes an input device 34 and a communication device 36 in addition tothe remote monitoring server 40 and the display device 32. Similar tothe display device 32, the input device 34 and the communication device36 are connected to the remote monitoring server 40.

The input device 34 is an interface for receiving input from the remoteoperator 50. Examples of the input device 34 include a touch panel, akeyboard, and a mouse. Also, if remote monitoring is performed forremotely driving the vehicle 10, the input device 34 includes a drivingoperation member for the remote operator 50 to operate (steer,acceleration, and decelerate) the vehicle 10.

The communication device 36 communicates with the outside of the vehicle10. The communication device 36 communicates with the on-boardinformation processing apparatus 20 via the communication network 8.

The remote monitoring server 40 is a computer for the remote monitoringcenter 30 or an aggregation a plurality of computers. The remotemonitoring server 40 includes at least one processor 41 (hereinaftersimply referred to as a processor 41) and at least one memory 42(hereinafter simply referred to as a memory 42) coupled to the processor41. The memory 42 includes a main storage and an auxiliary storage. Thememory 42 stores at least one program (hereinafter simply referred to asa program) executable by the processor 41 and various relatedinformation. When the processor 41 executes the program, various kindsof processing by the processor 41 are realized. The program may bestored in a main storage or in a computer-readable recording medium thatis an auxiliary storage.

The memory 42 includes at least two queues 42 a, 42 b. Plural pieces offirst data DT1 are stored in the first queue 42 a. The processor 41 putsthe latest first data DT1 in the first queue 42 a and extracts theoldest first data DT1 at the head from the first queue 42 a. The firstqueue 42 a stores the first data DT1 from i-th to i+j-th in the order ofthe time stamp TS.

Plural pieces of second data DT2 are stored in the second queue 42 b.The processor 41 puts the latest second data DT2 in the second queue 42b and extracts the oldest second data DT2 at the head from the secondqueue 42 b. The second queue 42 b stores the second data DT2 from i-thto i+j-th in the order of the time stamp TS. The processor 41 extractsthe first data DT1 and the second data DT2 whose time stamps TS are thesame time from the first queue 42 a and the second queue 42 b,respectively.

6. Processing Executed by Remote Monitoring System According to FirstEmbodiment

FIG. 9 is a block diagram showing processing executed by the remotemonitoring system 2 according to the first embodiment, morespecifically, processing executed by the processors 21 and 41 of theon-board information processing apparatus 20 and the remote monitoringserver 40. The flow of processing shown in this block diagram alsorepresents the remote monitoring method according to the firstembodiment of the present disclosure.

The processor 21 of the on-board information processing apparatus 20executes a time stamp generation process 211, an image compressionprocess 212, an object detection process 213, a first data transmissionprocess 214, and a second data transmission process 215. These processesare executed by the processor 21 when the program stored in the memory22 is executed by the processor 21.

In the time stamp generation process 211, the processor 21 generates atime stamp TS indicating the time at which an image IMG is acquired bythe camera 12. The generated time stamp TS is associated with the imageIMG and stored in the memory 22.

In the image compression process 212, the processor 21 reads out theimage IMG taken by the camera 12 from the memory 22. Then, the processor21 performs a predetermined compression process for the image IMG togenerate a compressed image CIMG. The generated compressed image CIMG isstored in the memory 22 associated with the time stamp TS.

In the object detection process 213, the processor 21 reads out theimage IMG taken by the camera 12 from the memory 22. The processor 21detects an object ahead of the vehicle 10 from the image IMG by using animage analysis technique such as pattern matching or deep learning. Theobject detection result ODR is stored in the memory 22 associated withthe time stamp TS.

In the first data transmission process 214, the processor 21 reads outthe first data DT1 including the compressed image CIMG and the timestamp TS from the memory 22. The processor 21 transmits the first dataDT1 to the remote monitoring server 40 using the communication device16.

In the second data transmission process 215, the processor 21 reads outthe second data DT2 including the object detection result ODR and thetime stamp TS from the memory 22. The processor 21 transmits the seconddata DT2 to the remote monitoring server 40 using the communicationdevice 16.

The processor 41 of the remote monitoring server 40 executes a firstdata reception process 411, a second data reception process 412, a sametime data extraction process 413, a compressed image restoration process414, and a superimposition display process 415. These processes areexecuted by the processor 41 when the program stored in the memory 42 isexecuted by the processor 41.

In the first data reception process 411, the processor 41 receives thefirst data DT1 from the on-board information processing apparatus 20using the communication device 36. The processor 41 stores the receivedfirst data DT1 in the first queue 42 a.

In the second data reception process 412, the processor 41 receives thesecond data DT2 from the on-board information processing apparatus 20using the communication device 36. The processor 41 stores the receivedsecond data DT2 in the second queue 42 b.

In the same time data extraction process 413, the processor 41 extractsthe compressed image CIMG and the object detection result ODR whose timestamps TS are the same time from the respective queues 42 a and 42 b intime series.

In the compressed image restoration process 414, the processor 41restores the compressed image CIMG extracted together with the objectdetection result ODR in a restoration method corresponding to thecompression method, and generates a restored image DIMG.

In the superimposition display process 415, the processor 41superimposes the object detection result ODR on the restored image DIMG,and displays the restored image DIMG on which the object detectionresult ODR is superimposed on the monitor screen 32 a of the displaydevice 32. Since the restored image DIMG and the object detection resultODR have the time stamps TS of the same time, there is no timedifference between the image and the object detection result that aredisplayed on the display device 32.

7. Configuration of Vehicle According to Second Embodiment

FIG. 10 is a block diagram showing a configuration of a vehicle 10equipped with an on-board information processing apparatus 20 accordingto the second embodiment. In FIG. 10 , the same elements as those of thevehicle 10 according to the first embodiment are denoted by the samereference numerals. Descriptions of the previously describedconfigurations and functions will be omitted or simplified.

Among the information acquired from the sensor signal of the on-boardsensor 14 (hereinafter referred to as sensor information) also includesinformation useful to present to the remote operator 50. Examples of thesensor information includes a LiDAR image acquired by imaging a LiDARsensor signal, and information about the distance and orientation of anobject acquired from sensor signals of a millimeter wave radar and anultrasonic sensor. They are crucial information for the remote operator50 to know the driving environment of the vehicle 10. In the secondembodiment, processing for acquiring the sensor information from sensorsignals is executed in the on-board information processing apparatus 20.In the second embodiment, the sensor information is transmitted from theon-board information processing apparatus 20 to the remote monitoringserver 40. The remote monitoring server 40 superimposes the sensorinformation on the restored image together with the object detectionresult, and displays the restored image on which the object detectionresult and the sensor information are superimposed on the display device32.

In the second embodiment, at least four storage areas 22 a, 22 b, 22 c,and 22 d are prepared in the memory 22. As in the first embodiment, thefirst storage area 22 a stores pre-processing data DT0 including animage IMG taken by the camera 12 and a time stamp TS indicating the timeat which the image IMG is acquired. The second storage area 22 a storesfirst data DT1 including a compressed image CIMG acquired by thecompression process for the image IMG and the time stamp TS associatedwith the image IMG before compression. The third storage area 22 cstores second data DT2 including an object detection result ODR acquiredby the object detection process for the image IMG and the time stamp TSassociated with the image IMG used for object detection.

The processor 21 stores third processing data (hereinafter, referred toas third data) DT3 in the fourth storage area 22 d of the memory 22. Thethird data DT3 includes sensor information SI acquired from a sensorsignal of the on-board sensor 14, and the time stamp TS indicating theacquisition time of the sensor signal. The processor 21 uses thecommunication device 16 to transmit the third data DT3 to the remotemonitoring server 40.

8. Configuration of Remote Monitoring Center According to SecondEmbodiment

FIG. 11 is a block diagram showing a configuration of the remotemonitoring center 30 that includes the remote monitoring server 40according to the second embodiment. In FIG. 10 , the same elements asthose of the remote monitoring center 30 according to the firstembodiment are denoted by the same reference numerals. Descriptions ofthe previously described configurations and functions will be omitted orsimplified.

In the second embodiment, the memory 42 includes at least three queues42 a, 42 b, and 42 c. As in the first embodiment, the first queue 42 astores the first data DT1 from i-th to i+j-th in the order of the timestamp TS. The second queue 42 b stores the second data DT2 from i-th toi+j-th in the order of the time stamp TS.

The third queue 42 b newly prepared in the second embodiment storesplural pieces of third data DT3. The processor 41 puts the latest thirddata DT3 in the third queue 42 c and extracts the oldest third data DT3at the head from the third queue 42 c. The third queue 42 c stores thethird data DT3 from i-th to i+j-th in the order of the time stamp TS.The processor 41 extracts the first data DT1, the second data DT2, andthe third data DT3 whose time stamps TS are the same time from the firstqueue 42 a, the second queue 42 b, and the third queue 42 c,respectively.

9. Processing Executed by Remote Monitoring System According to SecondEmbodiment

FIG. 12 is a block diagram showing processing executed by the remotemonitoring system 2 according to the second embodiment, morespecifically, processing executed by the processors 21 and 41 of theon-board information processing apparatus 20 and the remote monitoringserver 40. The flow of processing shown in this block diagram alsorepresents the remote monitoring method according to the secondembodiment of the present disclosure. In FIG. 10 , the same processes asthose executed in the first embodiment are denoted by the same referencenumerals. Descriptions of the previously described processes are omittedor simplified.

In addition to the processes executed in the first embodiment, theprocessor 21 of the on-board information processing apparatus 20according to the second embodiment executes a time stamp generationprocess 216, a sensor signal process 217, and a third data transmissionprocess 218. As with other processes, these processes are executed bythe processor 21 when the program stored in the memory 22 is executed bythe processor 21.

In the time stamp generation process 216, the processor 21 generates atime stamp TS indicating the time at which a sensor signal SS isacquired from the on-board sensor 14. The generated time stamp TS isstored in the memory 22 in association with the sensor signal SS.

In the sensor signal process 217, the processor 21 generates sensorinformation SI from the sensor signal SS. The sensor information SIincludes, for example, a LiDAR image acquired from a LiDAR sensorsignal, and range information and orientation information acquired fromsensor signals of a millimeter-wave radar and an ultrasonic sensor. Thegenerated sensor information SI is stored in the memory 22 inassociation with the time stamp TS associated with the sensor signal SS.

In the third data transmission process 218, the processor 21 reads outthe third data DT3 including the sensor data SI and the time stamp TSfrom the memory 22. The processor 21 transmits the third data DT3 to theremote monitoring server 40 using the communication device 16.

The processor 41 of the remote monitoring server 40 according to thesecond embodiment executes a third data reception process 416 inaddition to the processes executed in the first embodiment. In addition,the processor 41 executes a same time data extraction process 417 and asuperimposition display process 418, which are changed with respect tothe processes executed in the first embodiment. As with other processes,these processes are executed by the processor 41 when the program storedin the memory 42 is executed by the processor 41.

In the third data reception process 416, the processor 41 receives thethird data DT3 from the on-board information processing apparatus 20using the communication device 36. The processor 41 stores the receivedthird data DT3 in the third queue 42 c.

In the same time data extraction process 417, the processor 41 extractsthe compressed image CIMG, the object detecting result ODR, and thesensor information SI whose time stamps TS are the same time from therespective queues 42 a, 42 b, and 42 c in time series.

In the superimposition display process 418, the processor 41superimposes the object detection result ODR and the sensor informationSI on the restored image DIMG, and displays the restored image DIMG onwhich the object detection result ODR and the sensor information SI aresuperimposed on the monitor screen 32 a of the display device 32. Sincethe restored image DIMG, the object detection result ODR, and the sensorinformation SI have the time stamps TS of the same time, there is notime difference between the image, the object detection result, and thesensor information that are displayed on the display device 32.

What is claimed is:
 1. A remote monitoring system comprising: a cameramounted on a vehicle; an on-board information processing apparatusmounted on the vehicle and connected to the camera; and a remotemonitoring server connected to the on-board information processingapparatus via a communication network, wherein: the on-board informationprocessing apparatus is configured to execute: an object detectionprocess of detecting an object ahead of the vehicle from an imageacquired by the camera; a compression process of compressing the imageto generate a compressed image; a first data transmission process oftransmitting first data including the compressed image and an imageacquisition time of the image before compression; and a second datatransmission process of transmitting second data including an objectdetection result and an image acquisition time of the image used forobject detection; and the remote monitoring server is configured toexecute: a first data reception process of receiving the first data tostore the first data in a memory; a second data reception process ofreceiving the second data to store the second data in the memory; anextraction process of extracting the compressed image and the objectdetection result whose image acquisition times are a same time from thememory in time series; and a display process of superimposing anextracted object detection result on a restored image acquired byrestoring an extracted compressed image to display the restored image onwhich the extracted object detection result is superimposed on amonitoring screen.
 2. The remote monitoring system according to claim 1,wherein: the on-board information processing apparatus is configured tofurther execute a third data transmission process of transmitting thirddata including sensor information acquired from a sensor signal of anon-board sensor and a signal acquisition time of the sensor signal; theremote monitoring server is configured to further execute a third datareception process of receiving the third data to store the third data inthe memory; the extraction process includes extracting the compressedimage and the object detection result whose image acquisition times area same time from the memory in time series and extracting the sensorinformation whose signal acquisition time is a same time as the imageacquisition times from the memory in time series; and the displayprocess includes superimposing an extracted sensor information on therestored image together with the extracted object detection result todisplay the restored image on which the extracted object detectionresult and the extracted sensor information are superimposed on themonitoring screen.
 3. A method of remotely monitoring a vehicle using anon-board information processing apparatus and a remote monitoring serverconnected via a communication network, the method comprising: causingthe on-board information processing apparatus to execute: detecting anobject ahead of the vehicle from an image acquired by a camera mountedon a vehicle; compressing the image to generate a compressed image;transmitting first data including the compressed image and an imageacquisition time of the image before compression; and transmittingsecond data including an object detection result and an imageacquisition time of the image used for object detection; and causing theremote monitoring server to execute: receiving the first data to storethe first data in a memory; receiving the second data to store thesecond data in the memory; extracting the compressed image and theobject detection result whose image acquisition times are a same timefrom the memory in time series; and superimposing an extracted objectdetection result on a restored image acquired by restoring an extractedcompressed image to display the restored image on which the extractedobject detection result is superimposed on a monitoring screen.
 4. Themethod according to claim 3, further comprising: causing the on-boardinformation processing apparatus to further execute transmitting thirddata including sensor information acquired from a sensor signal of anon-board sensor and a signal acquisition time of the sensor signal;causing the remote monitoring server to further execute receiving thethird data to store the third data in the memory; extracting thecompressed image and the object detection result whose image acquisitiontimes are a same time from the memory in time series and extracting thesensor information whose signal acquisition time is a same time as theimage acquisition times from the memory in time series; andsuperimposing an extracted sensor information on the restored imagetogether with the extracted object detection result to display therestored image on which the extracted object detection result and theextracted sensor information are superimposed on the monitoring screen.5. A remote monitoring server connected to an on-board informationprocessing apparatus via a communication network, the remote monitoringserver comprising: at least one memory storing at least one program; andat least one processor coupled with the at least one memory, wherein theat least one program is configured to cause the at least one processorto execute: receiving first data including a compressed image generatedby compressing an image acquired by an on-board camera and an imageacquisition time of the image before compression from the on-boardinformation processing apparatus to store the first data in the at leastone memory; receiving second data including an object detection resultbased on the image and an image acquisition time of the image used forobject detection from the on-board information processing apparatus tostore the second data in the at least one memory; extracting thecompressed image and the object detection result whose image acquisitiontimes are a same time from the at least one memory in time series; andsuperimposing an extracted object detection result on a restored imageacquired by restoring an extracted compressed image to display therestored image on which the extracted object detection result issuperimposed on a monitoring screen.
 6. The remote monitoring serveraccording to claim 5, wherein the at least one program is configured tocause the at least one processor to further execute: receiving thirddata including sensor information acquired from a sensor signal of anon-board sensor and a signal acquisition time of the sensor signal tostore the third data in the at least one memory; extracting thecompressed image and the object detection result whose image acquisitiontimes are a same time from the memory in time series and extracting thesensor information whose signal acquisition time is a same time as theimage acquisition times from the at least one memory in time series; andsuperimposing an extracted sensor information on the restored imagetogether with the extracted object detection result to display therestored image on which the extracted object detection result and theextracted sensor information are superimposed on the monitoring screen.